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ABSTRACT 



A method of operating a digital video processor to continu- 
ously demultiplex, store and decode audio and video data 
streams and play back corresponding frames of audio and 
video. The processor further continuously demultiplexes, 
stores and decodes a first subpicture data stream and plays 
back first subpicture data iu synchronization with the play- 
back of frames of video data. PTS values associated with 
SPU's in the first subpicture data stream are continuously 
stored. The processor then detects a command instructing a 
transition from a first subpicture data stream to a second 
subpicture data stream and stores a PTS value associated 
with a first complete SPU of the second subpicture data 
stream so that it follows a PTS value associated with a last 
complete SPU in the first subpicture data stream. The 
processor then continuously stores PTS values associated 
with the SPU's in the second subpicture data stream. The 
first subpicture data stream is played back in synchroniza- 
tion with corresponding frames of video according to the 
stored PTS values associated with the SPU's of the first 
subpicture data stream; and the second subpicture data 
stream is played back in synchronization with corresponding 
frames of video according to the stored PTS values associ- 
ated with the SPU's of the second subpicture data stream. 
The above process is executed without interrupting the 
continuous decoding and playback of the audio and video 
data streams. 

20 Claims, 5 Drawing Sheets 
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SUBPICTURE STREAM CHANGE CONTROL command data block. Generally, the header is used to 

identify the general nature of the data, for example, the size 
of the subpicture unit and the location of the DCSQ com- 

BACKGROUND OF THE INVENTION mands. In the Subpicture format, the pixel data represents 

This invention relates to the digital processing of video 5 001(51 and contrast information and is compressed using 

data to be displayed on a television screen and more known compression techniques, for example, run length 

particularly, to the processing of a subpicture. compression. The DCSQ command block includes one or 

Almost all televisions manufactured today are capable of mo [ e °f commands that determine color and contrast 

interfacing with different sources of program materials, for j*** 11 * within the subpicture. In addition, DCSQ command 

example, a VCR, a digital versatile or video disk ("DVD") 10 %*™ y ,Jtft5£$ «nchide a Change Color-Contrast 

player, cable, DSS, etc., that provide audio signals for ("CHG_COLCON") command which functions to change 

creating sounds and associated video input signals for ere- ^ "tarnd contrast within the subpicture on a pixel by 

ating screen displays. Some of those sources provide digital pixe* oasis. 

audio and video input signals in accordance with the Moving Selected ones of the header blocks of the audio and video 

Picture Expert Group MPEG-2 audio/video digital compres- 15 data packets and all of the header blocks of the subpicture 

sion standard. Further, most televisions and/or their plug data packets include a presentation time stamp ("PTS") 

compatible program sources have user interactive capabili- value which is a time stamp that is applicable to that data 

ties with which a user may choose to have the programmed packet. The PTS value is a time reference to a system time 

source provide subpicture data for providing a display of clock mat was running during the creation or recording of 

captions, subtitles, karaoke or simple animation on the 20 me audio data - A similar system time clock 

screen along with the program material. Thus, contemporary ("STC") is also running during the playback of the audio and 

televisions and/or DVD systems preferably have the capa- video data > aild tf toe audio > video aod subpicture data are 

bility of processing compressed digital input signals repre- played back at the times represented by their presentation 

senting audio, video and subpicture data and providing ^me stamps, the audio, video and subpicture data will be 

digital output signals representing the desired sound, video 25 presented to the user in the desired synchronized manner, 

and subpicture images. Most often, those digital output Therefore, the PTS value is used to synchronize the presen- 

signals are converted to analog signals for use by known tation or playback of the audio, video and subpicture data, 

analog television display units. During the decoding of the audio data, it normally must 

The implementation of digital signal processing for pro- 3{) be decompressed, reconstructed and enhanced in a manner 

viding an audio, video and subpicture display from an consistent with the source of program material and the 

audio-video source of programmed material presents numer- capabilities of the sound reproduction system. In some 

ous design challenges that were not encountered in the prior applications, audio data packets may contain up to six 

processing of analog audio and video signals. For example, channels of raw audio data. Depending on the number of 

with digital signal processing, the audio, video and subpic- 35 channels the sound reproduction systems can reproduce, for 

ture signals are separated and are processed independently. example, from two to six, the sound reproduction system 

However, the playback of the audio, video and subpicture selectively uses the channels of raw audio data to provide a 

must be synchronized, so that there is a coordinated and number of channels of audio which are then stored in an 

coherent reproduction of the desired audio, video and sub- audio FIFO. 

picture provided from the source of program material. The decoding of the video data normally requires 
The source, for example, a video disk, preferably provides decompression, conversion of partial frames into full frames 
the audio and video data in respective data packets in an Jind the recognition of full frames. The decoding of subpic- 
"MPEG-2" format. Each of the audio, video and subpicture ture data requires the decompression of run length corn- 
data packets is received from the source of video material in pressed bit maps of subpicture data. Simultaneously with the 
a continuous data stream. Each packet of video data includes 45 decoding process, audio, video and subpicture data is being 
a header block followed by a data block. The data block may played back to the user, and in that playback, the frames of 
include any number, for example one to twenty, of frames of audio and video data are being output and the subpicture is 
video data that may include a full field of video data or be overlaid on top of the program video and the reconstructed 
a coded group of pictures that includes its own header block audio, video and subpicture must be synchronized in the 
identifying the picture type and display order. The header 50 playback process such that the audio, video and subpicture 
block for a video data packet includes control information, present a coordinated and coherent presentation, 
for example, the identity of the format of the video data, the As will be appreciated from the foregoing, demultiplexing 
type of compression, if used, picture size, display order, and the audio, video and subpicture data packets is a complex 
other global parameters. process of deconstructing the data packets and storing the 
Hie audio data packet has a header block that again 55 necessary decoding instructions as well as the content data 
identifies the format of the audio data with instructions itself to permit the decoding and playback of the data in a 
relating to how the audio data is to be decoded and processed synchronized manner. One such process, is described in a 
to provide desired enhancements, if applicable. Following copending U.S. patent application Ser. No. 08/901,090 
the header block, the audio data packet includes an audio entitled Method and Apparatus for Audio-Video 
data block that has any number of blocks or frames of audio go Synchronizing, filed on Jul. 28, 1997, now U.S. Pat. No. 
data, for example, from one to approximately twenty blocks. 5,959,684 and assigned to the assignee of the present 
Hie subpicture data may be provided in a data packet in application. U.S. patent application Ser. No, 08/901,090 is in 
one of several formats. For purposes of this description, it its entirety hereby incorporated by reference, 
will be assumed that the subpicture data is being provided in The interactive nature of current entertainment equipment 
a Subpicture format that is defined by the known DVD 65 presents additional problems in a synchronized playback of 
standard. The Subpicture format includes a header block, a audio, video and subpicture data. For example, subtitles in 
pixel data block, and a display control sequence ("DCSQ") a language different from the language of the audio are often 
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displayed in the playback of programmed media, for the PTS value of the first complete SPU in the second 

example, a movie on a video disk. A video disk may contain subpicture data stream to the stored PTS values of the 

subtitles in up to 32 different languages, and each subtitle complete SPlTs in the first subpicture data stream beginning 

language is stored as a subpicture data stream on the video with a most recendy stored PTS value. The processor then 

disk. The user has the capability of switching between any 5 identifies a closest stored PTS value of the complete SPU's 

of those 32 different languages at any point in time. Switch- in the first subpicture data stream that immediately precedes 

ing between different subpicture data streams often results in the PTS value of the first complete SPU in the second 

a temporary distortion in the subpicture display during the subpicture data stream. Thus, the processor provides a 

switching transition or a temporary loss of synchronization smoother transition from the first to the second subpicture 

with the video being played. 10 data streams by playing back all possible SPU's in the first 

Consequently, in a video system providing multiple subpicture data stream, 

streams of subpicture data, there is a need to provide an These and other objects and advantages of the present 

improved switching between different streams of subpicture invention will become more readily apparent during the 

data in response to randomly provided user requests. following detailed description taken in conjunction with the 

1 5 drawings herein. 



SUMMARY OF THE INVENTION 



BRIEF DESCRIPTION OF THE DRAWINGS 



The present invention provides a method and apparatus of 

a real time smooth transition between different streams of FIG. 1 is a schematic block diagram of a digital audio/ 

subpicture data without any delay or interference in the 2Q video decoder in accordance with the principles of the 

presentation of the main audio and video. The present present invention. 

invention has an advantage of utilizing only complete sub- FIG. 2 is a schematic block diagram of an ASIC device 

picture units in making a transition between first and second within the the audio/video decoder of FIG. 1. 

subpicture data streams. Thus, there is no partial subpicture FIG. 3 is a flow chart illustrating the steps of a portion of 

displays that may appear as video noise during the transition, ^ me demulti p lexing process executed by the demultiplexer in 

and there is no interruption to the presentation of the mam accordance with the principles of the present invention, 

audio ad video. The present invention utilizes all valid „ T _ , . , . . _ * 

subpicture unite in the first subpicture data stream prior to |?' 4 » » sc , he , m ^ c ^Tf T°l ? ^ PK ' 

switching to the subpicture units in the second subpicture daU to lUustrate the effect of a subpicture stream change 

data stream, thus playing back the optimum number of 30 comman 

multiplexed subpicture units in response to random user FIGS - 5A-5C are schematic drawings of different states 

requests to change from one subpicture data stream to of a subpicture PTS table in response to receipt of a 

another and coordinating to the greatest extent possible the subpicture stream change command, 

subpicture display to the video being played. FIG. 6 is a flow chart illustrating the steps in the process 

In accordance with the principles of the present invention 35 of a subpicture master control in accordance with the 

and the described embodiments, the invention provides a principles of the present invention, 

method of operating a digital video processor to conlmu- DETAILED DESCRIPTION OF THE 

ously demultiplex, store and decode audio and video data INVENTION 
streams and play back corresponding frames of audio and 

video. The processor further continuously demultiplexes, 40 Referring to FIG. 1, one embodiment of the present 

stores and decodes a first subpicture data stream and plays invention is for use in a DVD audio/video decoder 30 which 

back first subpicture data in synchronization with the play- includes a unit processor 31 with a program signal input 32 

back of frames of video data. PTS values associated with from a DVD drive. A central processing unit or host CPU 34 

SPU's in the first subpicture data stream are continuously which is programmed to process user commands from a 

stored. The processor then detects a command instructing a 45 control input device 33 operates a control system display 37 

transition from the first subpicture data stream to a second which displays information, menu selections and other 

subpicture data stream and stores a PTS value associated information to the user and which may or may not also 

with a first complete SPU of the second subpicture data function as an input device. An Application Specific Inte- 

stream so that it follows a PTS value associated with a last grated Circuit ("ASIC) 36, when provided with configura- 

complete SPU in the first subpicture data stream. The 50 tion and selection information by the host CPU 34, decodes 

processor then continuously stores PTS values associated the raw signal from signal input 32 for output to the video 

with the SPU's in the second subpicture data stream. The and audio presentation devices 38 and 40, respectively. A 

first subpicture data stream is played back in synchroniza- local system clock 41 and a buffer memory 42 are preferably 

tion with corresponding frames of video according to the connected to the ASIC 36. The buffer memory 42 is an 

stored PTS values associated with the SPU's of the first 55 in-line, sequential memory, such as dynamic random access 

subpicture data stream; and the second subpicture data or DRAM memory. 

stream is played back m syncrironizadon wim coiTesponding Components of the ASIC 36 are further described in 

frames of video according to the stored PTS values associ- commonly-assigned copending U.S. patent application Ser. 

ated with the SPU's of the second subpicture data stream. No. 08/865,749, now U.S. Pat. No. 6,012,137 entitled "SPE- 

Thus, processor provides a transition from the first to the & ciAL PURPOSE PROCESSOR FOR DIGITAL AUDIO/ 

second subpicture data streams without playing back any VIDEO DECODING", filed on May 30, 1997 and 

partial SPU's in either of the first or the second subpicture commonly-assigned, copending U.S. patent application Ser. 

data streams and without introducing any delay or interfer- No. 09/280,437 entitled "VARIABLE LENGTH 

ence in the presentation of the main audio and video. DECODER FOR DECODING DIGITALLY ENCODED 

In another embodiment of the invention, the above pro- 65 VIDEO SIGNALS", filed on even date herewith, which 

cessor determines a PTS value of a first complete SPU in the applications which are hereby incorporated by reference 

second subpicture data stream and successively compares herein in their entirety. A memory controller for use there- 
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with is disclosed in commonly-assigned, copending U.S. that connects externally of the ASIC 36 to the system clock 

patent application Ser. No. 08/846,590, now abandoned, 41, and has phase locked loop circuitry ("PLL") 82 within 

entitled "MEMORY ADDRESS GENERATION FOR the ASIC 36 used to time internal clock signals. 

DIGITAL VIDEO", filed on Apr. 30, 1997, which is hereby Audio, video and subpicture data packets are received and 

incorporated herein in its entirety. The above-referenced 5 demultiplexed continuously in independent parallel data 

U.S. patent applications describe an application specific streams. The decoding and playback of output frames of 

integrated circuit (ASIC) for performing digital video audio, video and subpicture data is also performed continu- 

processing, which is controlled by a reduced instruction set ously in parallel data streams independent of the demulti- 

CPU (RISC CPU). The RISC CPU controls computations plexing processes. Demultiplexing is a process that varies 

and operations of other parts of the ASIC to provide digital Q significantly in real time, depending on the nature of audio, 

video reception. Due to the limitations of the RISC CPU, a video and subpicture data being received. In addition, the 

task and stack manager procedure is required to monitor task number of video frames to be presented and their order of 

flags, prioritize task flags, manage subroutine calls (the presentation cannot be determined from the raw video data 

hardware does not support nesting of subroutine calls), and being received. The creation of video frames and their order 

provide virtual instruction memory management. A specific of presentation is a function of the decoding process and is 

processor of this kind is disclosed in commonly-assigned, determined primarily by the control data in the header 

copending U.S. patent application Ser. No. 08/866,419, now P orti °n of the video data packet. Similarly, the raw audio 

U.S. Pat. No. 5,928,321 entitled "TASK AND STACK data bem S received in the data P acket bears httle resem " 

MANAGER FOR DIGITAL VIDEO DECODING", filed on Wance to the audio data output and presented, and the frames 

u ~ n - nrv7 i_- f of audio data to be presented are created during the decoding 

May 30. 1997. which is hereby incorporated by reference „_ , , «, , . , j * • • j * 

3 r J 20 process of the audio data. The subpicture data is received in 

erein in i en e y. ... , a series of one or more data packets that include display 

Referring to FIG. 2, the ASIC 36 is a single integrated control sequence ("DCSQ") commands each of which has 

circuit chip that is logically divided into a number of its own start time ("STM") value. A subpicture unit ("SPU") 

components or functions. The ASIC 36 includes a memory is defined by the subpicture data occurring between subpic- 

controller and data bus 46, which provides a plurality of ^ ture data packets having a presentation time stamp ("PTS") 

two-way data flow connections. One of the two-way con- value. The intermediate subpicture data packets contain 

nections is to a static random access memory ("SRAM") 49 additional DCSQ command data. 

of the ASIC 36. Another of the two-way connections is to a pig. 3 is a flow chart illustrating the general operation of 

host interface unit 50 which connects externally with the me DMUX 52 of FIG. 2. At 202, the input 32 to the DMUX 

host CPU 34, and another is to the DRAM memory module 30 52 continuously receives a bit stream of data on input 32 

42 which is external to the ASIC 36. The ASIC 36 includes containing in random order, audio, video and subpicture data 

a demultiplexer or DMUX 52 which has an input connected packets. The header block of data is extracted at 204, and 

to the signal input 32 and an output delivering the received v |deo data packets are identified at 206. A video demulti- 

data to the bus 46. The DMUX 52 has a text output plexing interrupt is provided at 208 to the RISC 80 (FIG. 2); 

connected to a teletex processor 54, that is also provided on 35 and a t 210, the video data is sequentially stored in preferably 

the ASIC 36 for processing collateral information such as a contiguous variable length video data buffer, for example, 

closed caption script and other such data. a video first-in, first-out ("FIFO"), 43 in memory 42 (FIG. 

The ASIC 36 further includes an audio decoder 56, a 1) for use by the ASIC 3 6 during the video signal processing, 

video decoder 58 and a subpicture generating unit 62. The In a similar process, audio data packets are identified at 212, 

audio decoder 56 has an input side connected to the one of 40 and an audio demultiplexing interrupt is provided at 214 to 

the two-way data connections of the bus 46 and an output the RISC 80. At 216, the audio data is sequentially stored in 

connected to audio presentation subsystem 40, The video an audio data buffer, for example, an audio FIFO, 44 in 

decoder 58 receives video data via the two-way data con- memory 42 (FIG. 1). Subpicture data packets are identified 

nections of the bus 46, decodes and otherwise processes the at 218, and a subpicture demultiplexing interrupt is provided 

received video data, and sends the decoded and partially 45 at 220 to the RISC 80. At 222, the subpicture data is 

processed video picture data back through bus 46 to the sequentially stored in a subpicture data buffer, for example, 

DRAM memory 42. This processing preferably includes the a subpicture FIFO, 45 in memory 42 (FIG. 1). 

application of motion compensation calculations and the The demultiplexing process with respect to the audio, 

construction of B-picture fields from buffered I and/or P video and subpicture data continues in a manner described 

frames and received B-picture data. 50 in detail in the previously cited pending U.S. patent appli- 

The subpicture generating unit 62 generates local picture cation Ser. No. 08/901,090 entitled "Method and Apparatus 

information that includes control menus, display bar-graphs, for Audio-Video Synchronizing". As described therein, the 

captions, subtitles, karaoke or simple animation and other DMUX continuously provides audio, video and subpicture 

indicia used in interaction with the user. Normally, the PTS interrupts to the RISC 80. While the process of demul- 

during the decoding process, video data is supplied from the 55 tiplexing the audio, video and subpicture data continues, a 

DRAM 42 to a video blender 58. The video blender 58 process of decoding and playing back the audio, video and 

combines the program or main video with local video from subpicture frames is also running. In this process, there are 

the subpicture unit 62 and/or with teletex information from several different modes of operation by which the audio, 

the teletex processor 54. The output of the blender 58 is video and subpicture data are synchronized. For purposes of 

connected to the video presentation subsystem 38. eo the subpicture control, the audio data is selected as the 

The ASIC 36 is provided with a control bus (not shown) master and the video and subpicture frames are synchro- 

which is connected to the components in the ASIC 36. The nized thereto. The general process of synchronization and 

ASIC 36 is also provided with a Reduced Instruction Set the different synchronizing modes is explained in detail in 

Controller "(RISC) 80, which serves as the local CPU of the previously cited U.S. patent application Ser. No. 08/901, 

the ASIC 36. The RISC 80 controls the functions of the 65 090 entitled Method and Apparatus for Audio- Video Syn- 

components of the ASIC 36 through control data ports chronizing". The selection of the correct subpicture data to 

connected to the control bus. The RISC 80 has a clock input be displayed is performed during the vertical blanking time. 
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It should be noted that output audio frames can be of any As will be appreciated, the video data associated with the 

length in real time, and further, several audio frames may be subpicture streams is being demultiplexed and written into a 

associated with single video frame, or in contrast, a single subpicture FIFO 45 in the memory 42. Simultaneously, the 

audio frame may be presented during video produced by video data is also being read from the subpicture FIFO 45 for 

several video frames. However, it is required that the frames 5 decoding and display. Most often and desirably, there is a 

of audio and video be played back in a synchronized manner time difference between the steps of writing the subpicture 

to provide a coordinated and coherent presentation to the data into and thereafter, reading the subpicture data from, the 

user. To facilitate the coordination of the presentation of the FIFO 45. For example, as illustrated in FIG. 4, the stream 

frames of audio and video data, selected ones of the audio change command may occur after storing packet PIC, of 

and video data packets contain a PTS value, which is a time 10 SPU 5 but before demultiplexing packet PK J4 . As a result of 

reference to a system counter that was running during the the stream change command, the DMUX 52 will then be 

creation or recording of the audio and video data. A similar demultiplexing subpicture packet PK^ of subpicture unit 

system time clock ("STC) is maintained and clocked in real SPU A . However, at the time the stream change command 

time, for example, in register 86, by the DMUX 52; and occurs, the decoding process may be reading the subpicture 

during the demultiplexing process, audio, video and subpic- 15 unit SPUj associated with PTSj from the subpicture FIFO 

ture PTS values are stored in respective PTS tables. During 45. 

the standard decoding and playback, the audio and video One method of executing the transition from stream 1 to 

PTS values in the tables are compared to the STC times; and stream 2 is to sequentially decode all of the subpicture data 

when a PTS value is equal to or less than the STC time, the packets up through packet PK J3 of SPU 5 and then jump 

respective audio, video and subpicture data is read from 20 immediately to the packet PK^, of SPU A . While such a 

memory, decoded and played at a time and in a sequence that process has the advantage of requiring no special handling 

conforms to the how the data was recorded on the DVD. of the subpicture data, it has the significant disadvantage of 

With respect to the subpicture data, the RISC 80 decodes attempting to show incomplete subpicture units which 

the DCSQ commands in the subpicture during the vertical results in substantial distortion and a garbled subpicture 

blanking period, that is, with each vertical synchronization 25 display at the point of the transition from stream 1 to stream 

period ("fid"). The decoding of the subpicture DCSQ com- 2. Further, because the PTS values in stream 2 do not exactly 

mands is initiated by the RISC 80 when the PTS values in follow the PTS values in stream 1, the subpicture of stream 

each SPU and the STM values associated with each DCSQ 2 may be out of synchronization with the video. A second 

command in intermediate subpicture data packets is less approach is to finish the decoding of SPV 1 and then jump 

than the time values in the STC. Upon determining the 30 immediately to decode SPU B . While that method has less 

appropriate DCSQ command to be executed, the RISC 80 distortion than the method previously described, it has the 

provides first command data, for example, subpicture loca- disadvantage of potentially not using all of the subpicture 

tion data and color and contrast data to the subpicture data from stream 1 that has been demultiplexed and stored, 

generator 62 and further initiates the transfer of subpicture Abetter method of switching from subpicture stream 1 to 

pixel data from memory 42 to the subpicture generator 62. 35 subpicture stream 2 is to display complete SPU's of stream 

The RISC 80 further causes the pixel data for the video to 1 up to the PTS of the first complete SPU in stream 2. For 

be sequentially provided from the memory 42 to the video example, if the VTS B value of SPU B is greater than the value 

blender 58. Simultaneously therewith, the subpicture gen- of PTS 3 but less than the value of PTS 4 , then SPU 5 should 

erator 62 provides, if appropriate, subpicture pixel data to be read and decoded after SPU 3 . That method provides the 

the video blender 58. The video blender 58 utilizes a known 40 optimal transition from subpicture stream 1 to subpicture 

process, for example, a mixing process, to mix the subpic- stream 2 and best coordinates the display of subpicture data 

ture pixels with the video pixels from memory 42 and with the video being played. 

produce the desired mixed or blended video data. The The above preferred subpicture stream change method is 

blended video data is then encoded in accordance with a implemented by the RISC 80 creating a subpicture PTS table 

desired standard, for example, a NTSC or PAL standard; and 45 in memory 42 as illustrated in FIG. 5A. The subpicture PTS 

thereafter, the encoded video data is converted to an analog table is a buffer that stores successive PTS values associated 

signal and displayed on a display unit 38. with successive SPU's. With each PTS value, the RISC 80 

FIG. 4 illustrates two streams of subpicture data. The also stores an associated base address which identifies the 

subpicture data of stream 1 is comprised of a series of location of the associated SPU in the memory 42. Thus, the 

subpicture units SPU 0 , SPU a , SPU 2 , etc. Each subpicture so subpicture PTS table values are used to determine when an 

unit includes one or more packets of subpicture data and an SPU is to be played and where the SPU data is located in 

associated PTS value. Under normal conditions, the packets memory. Every time the DMUX 52 receives a subpicture 

of subpicture data are decoded and displayed in the order in PTS value, it sends a subpicture PTS interrupt to the RISC 

which they are processed by the DMUX. However, as 80, and the subroutine illustrated in FIG. 6 is executed. The 

previously described, the user has the capability of using the 55 subroutine first at 502, tests whether a stream change com- 

input device 33 (FIG. 1) to command a transition from one mand has been received. Assuming that the DMUX 52 has 

subpicture display to another, for example, to display sub- detected PTS 0 and there is no stream change command, the 

titles in a different language. That command is detected by subroutine then reads the PTS write pointer at 504, and the 

the host CPU 34 and passed on to the ASIC 36 which current PTS value, that is, PTS 0 , is stored in the subpicture 

changes the identity of the subpicture data stream in the 60 PTS table of FIG. 5 A- Thereafter, at 508, the subroutine of 

DMUX 52. Such a user initiated switching point is repre- FIG. 6 reads the current base address from the PTS table, 

sented in FIG. 4 by the stream change command line and at 510, the byte count from the DMUX 52 is added to 

intersecting both streams 1 and 2. In responding to the the base address and the sum stored in the PTS table. In the 

stream change command, the system must execute a tran- present case, PTS 0 , is the first PTS; and therefore, the byte 

sition of the subpicture data from stream 1 to stream 2 as 65 count is 0. The subroutine then increments the write pointer 

smoothly as possible with minimum distortion while main- at 512 so that it is now located at the second location in the 

taining synchronization with the video. subpicture PTS table of FIG. 5 A. 
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When the DMUX 52 detects the next PTS, that is, PTS 1( 
another PTS interrupt is provided from the DMUX to the 
RISC 80, and the subroutine of FIG. 6 is again executed. As 
previously described, the subroutine at 506 writes the value 
of PTS j, in the second location of the PTS table. Thereafter, 
at 508, the current base address, that is, BAq, is read from the 
PTS table. At 510, the byte counts BQ-BC3 detected by the 
DMUX 52 in processing the subpicture data packets 
PKj-PKs arc added to the current base address, and the sum 
is stored in the PTS table as base address BA r Thereafter, 
the write pointer is incremented to the third table location at 
512. The subroutine of FIG. 6 iterates as described above in 
processing subpicture data stream 1 through subpicture unit 
SPU 4. 

In summary, as the DMUX 52 processes the packets 
within a subpicture unit, it detects the byte count for each 
packet. Thus, the PTS table is used to store the PTS value for 
a subpicture unit and in addition, the base address for that 
subpicture unit, that is, the address within the subpicture 
FIFO 45 at which the video data associated with a packet is 
stored. Therefore, upon playback, when the presentation 
time stamp for a subpicture unit is equal to the system time 
clock, the base address which is a summation of the byte 
counts may be used to direct the read pointer to read that 
subpicture unit video data from subpicture FIFO 45. 

During the processing of SPU 5 , the DMUX 52 detects the 
stream change command initiated by the user As a result, 
after the DMUX 52 processes PK 12 and PK 13 of SPU 3 , it 
will then receive and process PK^, and PK^ of SPU A . The 
net result is that the system has demultiplexed video data for 
two incomplete subpicture units SPU 5 and SPU A . 
Consequently, if the system attempted to display those 
subpicture units, the video would appear as being incom- 
plete and discontinuous during the subpicture display. 
Therefore, it is desirable that SPU 5 and SPU^ not be 
displayed and be skipped in the playback process. Further, 
it is desirable that SPU^ be displayed at the appropriate time 
with respect to the other stored subpicture units 
SPU 0 -SPU 4 . Thus, the system must determine the value of 
PTS^ with respect to the values of the PTS 0 -PTS 4 ; and if 
PTS^ is less than PTS 4 , PTS 4 should be skipped. 

It should be noted that upon the DMUX 52 detecting a 
stream change command, the DMUX 52 will have also 
determined the value of PTS^ and will have accumulated the 
sum of the byte counts BC 12 , BC 13 , BC^, BC^. Further, at 
the time of detecting the stream change command, the 
subpicture PTS table will be filled as indicated in FIG. 5A 
with the write pointer at the seventh table location, but the 
values for the seventh table location yet to be determined. 
Upon detecting a stream change command at 502, the 
subroutine of FIG. 5 then at 514, determines the relative 
locations and separation of the PTS read and write pointers. 
If at 516, the read and write pointers are determined to be at 
immediately adjacent locations within the PTS table, then 
there can be no optimizing of the stream change process. 
There can be no interruption to the display of video data, and 
therefore, when the pointers are immediately adjacent each 
other, there is no opportunity to modify the PTS table. Under 
those circumstances, the FTS B and the associated detected 
byte counts through BC22 will be written into the seventh 
location of the table and the incomplete subpicture units 
SPU 5 and SPU A will be played with the resulting disconti- 
nuities. 

Under normal circumstances, the PTS write pointer is 
significantly ahead of the read pointer in the subpicture PTS 
table, and therefore, the subpicture optimization can occur. 
Assume the write pointer is at the seventh table location and 
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the read pointer is at the second table location, thereby 
providing a significant separation between the read and write 
pointers. The write pointer is decremented at 518 to the sixth 
table location; and at 520, a reference base address location, 

5 that is, BAj for PTS 5 , is saved. Once again, the location of 
the read pointer with respect to the write pointer is tested at 
522, 524; and if it is determined that the read and write 
pointers are immediately adjacent each other, then the write 
pointer is incremented to the sixth table location and the 

10 saved base address BA 3 is saved in its original location in 
the table. Thereafter, at steps 504-510, PTS^ is written into 
the sixth table location, and the byte counts are added to the 
base address BA^. The resulting subpicture PTS table is 
illustrated in FIG. 5B. Thus, on playback, the subpicture 

15 stream 1 is played through subpicture unit SPU 4 including 
PK 10 and PK n . Thereafter, upon PTS B being equal to the 
system time clock, the read pointer in the video data memory 
will jump to start of SPU^ and begin playing video data 
associated with Pl^. The net effect of decrementing the 

2P write pointer twice at 518 and 528 is to skip SPU 5 , and 
SPU A , thereby eliminating the playback of incomplete 
SPU's that would present a garbled display of subpicture. 

If, at 524, it is determined that the pointers are not 
immediately adjacent each other, the write pointer is again 

25 decremented at 528 to the fifth table location PTS 4 . 
Thereafter, at 530, 532, the value of PTS 4 is read from the 
table and compared to the current PTS, that is, PTS^. If at 
532, the subroutine detects that PTS B is not less than PTS 4 , 
the process increments the write pointer to the sixth table 

30 location, and the previously saved base address BA5 is 
written thereto. The subroutine then in steps 504-510, 
completes the filling of the sixth table location to provide a 
stream change in PTS table as illustrated in FIG. 5B. 
If at 532, the subroutine determines that PTS^ is less than 

35 PTS 4 , the subroutine at 522, 524, tests for the location of the 
PTS table read pointer, and if it is not immediately adjacent 
the write pointer, the write pointer is again decremented at 
528 to the fourth table location. Thereafter, at 532, if it is 
determined that PTS^ is not less than PTS 3 , the write pointer 

40 is incremented at 526 and the saved reference base address 
BA^ is written into the fifth table location. The routine at 
504-510 continues to store PTS^ at the fifth table location 
along with its associated byte count. The write pointer is 
incremented at 512 and the net result is a subpicture PTS 

45 table as illustrated in FIG. 5C. With that table, after the 
subpicture SPU 3 is played, when the PTS^ equals the system 
time clock, the subpicture unit SPUg will be played. Thus, 
subpicture units SPU 4 , SPU 5 , and SPU A are skipped. 
In use, with the above system, only complete subpicture 

50 units are displayed, and further, if the stream change is 
detected, all of the multiplexed and stored subpicture units 
are tested against the first subpicture unit in the new stream; 
and the new stream is played at the appropriate time so that 
it is synchronized with the video. Further, the optimal 

55 number of SPU's are displayed without having to manipu- 
late the subpicture data in the video FIFO 45 in DRAM 42 
where- all of the demultiplexed subpicture data is stored. 
However, by detecting the relative PTS values between the 
first complete SPU of the next stream and the complete 

60 SPU's of the old stream and modifying the subpicture PTS 
table, the optimal subpicture data is read from the subpicture 
FIFO 45. Further, the above process is executed with the 
simultaneous real time playback of the audio and video, and 
the transition from one subpicture stream change to another 

65 occurs without introducing any delay, noise or interference 
with the continuous real time playback of the main audio and 
video. 
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It should be further noted that in the subpicture PTS tables 
of FIGS. 5A-5C, the read pointer is shown to be at a 
constant position. As will be appreciated, the write and read 
pointers are constantly moving as a function of the inde- 
pendent and respective multiplexing and decoding pro- 
cesses. Therefore, the separation between the read and write 
pointers is also constantly changing. 

While the invention has been illustrated by the description 
of a preferred embodiment and while the embodiment has 
been described in considerable detail, there is no intention to 
restrict nor in any way limit the scope of the amended claims 
to such detail. Additional advantages and modifications will 
readily appear to those who are skilled in the art. For 
example, at 532 of FIG. 6, the subroutine requires that the 
current PTS be less than the PTS table value in order for the 
subroutine to test the next lower value in the PTS table. In 
other words, if the first PTS value in the second subpicture 
data stream is equal to the PTS value of the first subpicture 
data stream stored in the table, the table value is given 
priority; and the SPU in the first subpicture data stream 
associated with the stored PTS value will be played back. 

As will be appreciated, action can be taken with respect 
to the current PTS in the second subpicture data stream in 
response to other relationships to the stored PTS table 
values. For example, the test at 532 in FIG. 6 may be 
changed to test whether the current PTS in the second 
subpicture data stream is equal to or less than the stored PTS 
table value. In that embodiment, the first PTS in the second 
subpicture data stream is given priority and the stored PTS 
value from the first subpicture data stream is replaced and 
not played back. In playing back the subpicture, generally, 
it is not particularly important that a subpicture be precisely 
identified with a particular frame of video. Therefore, there 
is a certain latitude in choosing exactly when to play back 
the subpicture; and a particular relationship used is generally 
a matter of design choice. For ease of implementation, it is 
more important, that whatever relationship is chosen, that it 
be used for all such comparisons. 

Therefore, the invention in its broadest aspects is not 
limited to the specific details shown and described. 
Consequently, departures may be made from the details 
described herein without departing from the spirit and scope 
of the claims which follow. 

What is claimed is: 

1. A method of operating a digital video processor in 
response to audio, video and subpicture data streams, the 
digital video processor including a CPU and a memory and 
the subpicture data streams comprising a serial stream of 
subpicture units ("SPU V) wherein a start of each of the 
SPU's has a presentation time stamp ("PTS") associated 
therewith, the method comprising: 

continuously demultiplexing, storing and decoding the 
audio and video data streams and playing back corre- 
sponding frames of audio and video; 
continuously demultiplexing, storing and decoding a first ss 
subpicture data stream and playing back first subpicture 
data in synchronization with the playback of frames of 
video data; 

continuously storing PTS values associated with SPU's in 
the first subpicture data stream; go 

detecting a command instructing a transition form the first 
subpicture data stream to a second subpicture data 
stream; 

storing a PTS value associated with a first complete SPU 
of the second subpicture data stream so that it follows 65 
a PTS value associated with a last complete SPU in the 
first subpicture data stream; 



continuously storing PTS values associated with the 

SPU's in the second subpicture data stream; 
playing back the first subpicture data stream in synchro- 
nization with corresponding frames of video according 
to the stored PTS values associated with the SPU 'of the 
first subpicture data stream; and 
playing back the second subpicture data stream in syn- 
chronization with corresponding frames of video 
according to the stored PTS values associated with the 
SPU's of the second subpicture data stream, thereby 
providing a transition from the first to the second 
subpicture data streams without playing back any par- 
tial SPU's in either of the first or the second subpicture 
data streams and without any interruption to the con- 
tinuous demultiplexing, storing and decoding of the 
audio and video data streams. 

2. A method of claim 1 further comprising continuously 
storing with each stored PTS value associated with SPU's in 
both the first and the second subpicture data streams a 
memory location of the current PTS value being stored. 

3. A method of claim 1 further comprising continuously 
storing with each stored PTS value associated with SPU's in 
both the first and the second subpicture data streams a 
number of bytes from an immediately preceding memory 
location storing a PTS value to a memory location of the 
current PTS value being stored. 

4. A method of claim 1 further comprising: 
determining a PTS value of a first complete SPU in the 

second subpicture data stream; 
successively comparing the PTS value of the first com- 
plete SPU in the second subpicture data stream to the 
stored PTS values of the complete SPU's in the first 
subpicture data stream beginning with a most recently 
stored PTS value; and 
identifying a closest stored PTS value of the complete 
SPU's in the first subpicture data stream that immedi- 
ately precedes the PTS value of the first complete SPU 
in the second subpicture data stream. 

5. A method of claim 1 further comprising detecting a PTS 
value of the first complete SPU in the second subpicture data 
stream being greater than the stored PTS values of the 
complete SPU's in the first subpicture data stream to identify 
the closest stored PTS value of the complete SPU in the first 

45 picture data stream. 

6. A method of claim 1 further comprising detecting a PTS 
value of the first complete SPU in the second subpicture data 
stream being greater than or equal to the stored PTS values 
of the complete SPU's in the first subpicture data stream to 
identify the closest stored PTS value of the complete SPU in 
the first picture data stream. 

7. A method of operating a digital video processor in 
response to audio, video and subpicture data streams, the 
digital video processor including a CPU and a memory and 
the subpicture data streams comprising a serial stream of 
subpicture units ("SPU's") wherein a start of each of the 
SPU's has a presentation lime stamp ("PTS") associated 
therewith, the method comprising: 

continuously demultiplexing, storing and decoding the 
audio and video data streams and playing back corre- 
sponding frames of audio and video; 
continuously demultiplexing, storing and decoding a first 
subpicture data stream and playing back first subpicture 
data in synchronization with the playback of frames of 
video data; 

continuously storing PTS values associated with SPU's in 
the first subpicture data stream; 
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detecting a command instructing a transition from the first 
subpicture data stream to a second subpicturc data 
stream; 

determining a PTS value of a first complete SPU in the 

second subpicture data stream; 
comparing the PTS value of the first complete SPU in the 

second subpicture data stream to the stored PTS values 

of the complete SPU's in the first subpicture data 

stream; 

identifying a closest stored PTS value of the complete 
SPU's in the first subpicture data stream that immedi- 
ately precedes the PTS value of the first complete SPU 
in the second subpicture data stream; 

storing the PTS value associated with the first complete 15 
SPU of the second subpicture data stream so that it 
follows the closest stored PTS value associated with a 
last complete SPU in the first subpicture data stream; 

continuously storing other PTS values associated with the 
SPU's in the second subpicture data stream; 20 

playing back the SPU's in the first subpicture data stream 
up through the SPU having the closest PTS value in 
synchronization with corresponding frames of video 
according to the stored PTS values associated with the 
SPU's of the first subpicture data stream and up 25 
through the closest PTS value; and 

playing back the second subpicture data stream starting 
with the first complete SPU in synchronization with 
corresponding frames of video according to the stored 
PTS values associated with the SPU's of the second 30 
subpicture data stream, thereby providing a smooth 
transition from the first to the second subpicture data 
streams without interfering with the continuous 
demultiplexing, storing and decoding the audio and 
video data streams and playing back of corresponding 35 
frames of audio and video. 

8. A method of operating a digital video processor includ- 
ing a CPU and a memory to change from a first subpicture 
data stream to a second subpicture data stream, each of the 
first and second subpicture data streams comprising a 40 
respective serial stream of subpicture units ("SPU's"), a start 

of each of the SPU's having a presentation time stamp 
("PTS") associated therewith, the method comprising: 
creating a subpicture PTS table in the memory for storing 
successive PTS values for successive SPU's in the first 45 
subpicture data stream and respective associated byte 
counts including a number of bytes between successive 
SPU's; 

detecting a command requesting a change from the first to 
the second subpicture data stream; 

comparing a first PTS value in the second subpicture data 
stream to the PTS values of the first subpicture data 
stream stored in the subpicture PTS table; 

storing the first PTS value in the second subpicture data 55 
stream in a subpicture PTS table location having a first 
stream PTS value greater than the first PTS value in the 
second subpicture data stream; and 

storing in a location in the subpicture PTS table associated 
with the first PTS value in the second subpicture data 60 
stream a byte count representing the sum of all of the 
byte counts in the first and second subpicture data 
streams demultiplexed up to the first PTS value in the 
second subpicture data stream. 

9. The method of claim 8 further comprising storing the 65 
first PTS value in the second subpicture data stream in a 
subpicture PTS table location having a first stream PTS 



50 



value greater than or equal to the first PTS value in the 
second subpicture data stream. 

10. The method of claim 8 further comprising: 
receiving a PTS interrupt from a demultiplexer in 

response to the demultiplexer processing a current 
SPU; 

testing whether the command requesting a change from 
the first to the second subpicture data stream has been 
received; 

writing a current PTS value associated with the current 
SPU in the subpicture PTS table in response to deter- 
mining that the command has not been received; 

determining a current byte count including a number of 
bytes from a start of a prior SPU to the first SPU; 

reading a base address in byte counts from the subpicture 
PTS table associated with the prior SPU, the base 
address identifying a location of subpicture data in the 
memory associated with the prior SPU; 

combining the current byte count with the base address to 
form a current base address identifying a location of 
subpicture data in memory associated with the current 
SPU; and 

writing the current base address into a location in the 
subpicture PTS table associated with the current PTS 
value of the current SPU. 

11. The method of claim 10 wherein the subpicture PTS 
table has a read pointer and a write pointer, the method 
further comprising: 

comparing the locations of the read and write pointers in 
the subpicture PTS table in response to determining 
that the command requesting a change from the first to 
the second subpicture data stream has been received; 

decrementing the write pointer one location within the 
subpicture PTS table in response to the read and write 
pointers not being at adjacent locations within the 
subpicture PTS table; 

saving a reference base address associated with a current 
location of the write pointer; 

comparing again the locations of the read and write 
pointers in the subpicture PTS table; 

decrementing the write pointer another location within the 
subpicture PTS table in response to the read and write 
pointers again not being at adjacent locations within the 
subpicture PTS table; and 

comparing another PTS value from the first subpicture in 
response stored in the other location in the subpicture 
PTS table with the first PTS value in the second 
subpicture data stream. 

12. The method of claim 11 further comprising: 
writing the reference base address at the other location of 

the subpicture PTS table in response to the value of the 
current PTS not being less than the first PTS value in 
the second subpicture data stream; 

incrementing the write pointer to the one location in the 
subpicture PTS table; and 

writing in the one location of the subpicture PTS table, the 
value of the first PTS value in the second subpicture 
data stream and a base address value equal to a sum of 
the reference base address and a number of byte counts 
detected from the reference base address to the first 
PTS value in the second subpicture data stream. 

13. The method of claim 12 comprising: 
comparing again the locations of the read and write 

pointers in the subpicture PTS table in response to the 
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value of the first PTS value in the second subpicture 
data stream being less than the other PTS value; 

decrementing the write pointer a further location within 
the subpicture PTS table in response to the read and 
write pointers again not being at adjacent locations 
within the subpicture PTS table; and 

comparing a further PTS value from the first subpicture 
data stream stored in the other location in the subpic- 
ture PTS table with a value of the first PTS value in the 
second subpicture data stream. 

14. The method of claim 13 further comprising iterating 
the methods of claims 5 and 6 for all PTS values in the 
subpicture PTS table that are greater than the first PTS value 
in the second subpicture data stream. 

15. The method of claim 14 further comprising iterating 
the methods of claims 5 and 6 for all PTS values in the 
subpicture PTS table that are greater than or equal to the first 
PTS value in the second subpicture data stream. 

16. A method of operating a digital video processor 
including a CPU and a memory to change from one sub- 
picture data stream to a second subpicture data stream, each 
of the first and second subpicture data streams comprising a 
respective serial stream of subpicture units ("SPU's"), a start 
of each of the SPU's being defined by a presentation time 
stamp ("PTS") associated therewith, the method comprising: 

creating a subpicture PTS table in the memory for storing 
successive PTS values for successive SPU's in the first 
subpicture data stream and respective associated byte 
counts including a number of bytes between successive 
SPU's; 

detecting a command requesting a change from the first to 
the second subpicture data stream; 

comparing the locations of the read and write pointers in 
the subpicture PTS table in response to determining 
that the command has been received; 

decrementing the write pointer in the subpicture PTS 
through successive locations in the subpicture PTS 
table in response to the read and write pointers not 
being in adjacent locations in the subpicture PTS table ^ 
with each decrement of the write pointer; 

with each decrement of the write pointer, comparing a first 
PTS value in the second subpicture data stream to a 
PTS value associated with a current location of the 
write pointer; 45 

storing the first PTS value in the second subpicture data 
stream in the subpicture PTS table location having a 
first stream PTS value greater than the first PTS value 
in the second subpicture data stream; and 

storing in a location in the subpicture PTS table associated 50 
with the first PTS value in the second subpicture data 
stream a byte count representing the sum of all of the 
byte counts in the first and second subpicture data 
streams demultiplexed up to the first PTS value in the 
second subpicture data stream. 

17. The method of claim 16 further comprising storing the 
first PTS value in the second subpicture data stream in the 
subpicture PTS table location having a first stream PTS 
value greater than or equal to the first PTS value in the 
second subpicture data stream. 

18. A method of operating a digital video processor 
including a CPU and a memory to change from one sub- 
picture data stream to a second subpicture data stream, each 
of the first and second subpicture data streams comprising a 
respective serial stream of subpicture units ("SPU's"), each 65 
of the SPU's having a presentation time stamp ("PTS") 
associated therewith, the method comprising: 
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demultiplexing the first subpicture data stream including 
first stream PTS values; 

storing successive first stream PTS values in successive 
locations in a PTS table in the memory; 

storing in locations in the PTS table associated with each 
of the first stream PTS values a respective byte count 
including a sum of byte counts between successive 
SPU's in the first subpicture data stream; 

detecting a command requesting a change from the first to 
the second subpicture data stream; 

demultiplexing the second subpicture data stream includ- 
ing second stream PTS values; 

comparing a first second stream PTS value to first stream 
PTS values stored in the PTS table; 

storing the first second stream PTS value in a PTS table 
location having a first stream PTS value greater than 
the first second stream PTS value; and 

storing in a location in the PTS table associated with the 
first second stream PTS value stream a byte count 
including a sum of all of the byte counts in the first and 
second subpicture data streams demultiplexed up to the 
first second stream PTS value. 

19. The method of claim 18 further comprising storing the 
first second stream PTS value in a subpicture PTS table 
location having a first stream PTS value greater than or equal 
to the first second stream PTS value. 

20. A digital video processor being responsive to audio, 
video and subpicture data streams, the subpicture data 
streams comprising a serial stream of subpicture units 
("SPU's") wherein a start of each of the SPU's has a 
presentation time stamp ("PTS'*) associated therewith, the 
processor comprising: 

a demultiplexer for continuously demultiplexing the 
audio and video data streams and subpicture data 
streams; 

a memory electrically connected to the demultiplexer for 
storing demultiplexed audio and video data and demul- 
tiplexed subpicture data, the memory further storing 
PTS values associated with SPU's in a first subpicture 
data stream; 

a CPU electrically connected to the memory and the 
demultiplexer for detecting a command requesting a 
transition from a first subpicture data stream to a 
second subpicture data stream, the CPU controlling the 
demultiplexer and the memory to store a PTS value 
associated with a first complete SPU of the second 
subpicture data stream so that it follows a PTS value 
associated with a last complete SPU in the first sub- 
picture data stream and thereafter continuously storing 
PTS values associated with the SPU's in the second 
subpicture data stream; 

a decoder electrically connected to the CPU and the 
memory for decoding the demultiplexed audio and 
video data and continuously playing back correspond- 
ing frames of audio and video; 

a subpicture generator and blender electrically connected 
to the CPU, the decoder and the memory and respon- 
sive to the demultiplexed subpicture data streams for 
playing back the first subpicture data stream in syn- 
chronization with corresponding frames of video 
according to the stored PTS values associated with the 
SPU's of the first subpicture data stream and also 
playing back the second subpicture data stream in 
synchronization with corresponding frames of video 
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according to the stored PTS values associated with the data streams and without any interruption to the con- 
SPU's of the second subpicture data stream, thereby tinuous demultiplexing, storing and decoding of the 
providing a transition from the first to the second audio and video data streams, 
subpicture data streams without playing back any par- 
tial SPlTs in either of the first or the second subpicture ***** 
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Line 27, reads "audio ad video" and should read - audio and video 
Column 4, 

Line 66, reads "which applications which are" and should read ~ which applications 
are --. 

Column 5, 

Line 54, reads "Normally, the during the" and should read ~ Normally, during the --. 
Line 64, reads "Controller "(RISC') 80," and should read Controller ("RISC") 80, --. 

Column 6, 

Line 65, reads "entitled Method and" and should read -- entitled "Method and --. 
Column 7, 

Line 22, reads "conforms to the how the" and should read -- conforms to how the — . 
Column 8, 

Line 9, reads "PK 3 " and should read » PK 13 --. 
Column 11, 

Line 11, reads "restrict nor in any" and should read -- restrict or in any --. 
Line 11, reads "scope of the amended claims" and should read -- scope of the 
appended claims «. 

Line 6 , reads "transition form the first" and should read - transition from the first -. 
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